Realistic Face Animation From Sparse Stereo Meshes

URL : http://spitswww.uvt.nl/Fsw/Psychologie/AVSP2007/papers/bergerAVSP.pdfInternational audienceBeing able to produce realistic facial animation is crucial for many speech applications in language learning technologies. For reaching realism, it is necessary to acquire and to animate dense 3D models of the face. Recovering dense models is often achieved using stereovision techniques. Unfortunately, reconstruction artifacts are common and are mainly due to the difficulty to match points on untextured areas of the face between images. In this paper, we propose a robust and fully automatic method to produce realistic dense animation. Our input data are a dense 3D mesh of the talker obtained for one viseme as well as a corpus of stereo sequences of a talker painted with markers that allows the face kinematics to be learned. The main contribution of the paper is to transfer the kinematics learned on a sparse mesh onto the 3D dense mesh, thus allowing dense facial animation. Examples of face animations are provided which prove the reliability of the proposed method

OA-SLAM: Leveraging Objects for Camera Relocalization in Visual SLAM

In this work, we explore the use of objects in Simultaneous Localization and Mapping in unseen worlds and propose an object-aided system (OA-SLAM). More precisely, we show that, compared to low-level points, the major benefit of objects lies in their higher-level semantic and discriminating power. Points, on the contrary, have a better spatial localization accuracy than the generic coarse models used to represent objects (cuboid or ellipsoid). We show that combining points and objects is of great interest to address the problem of camera pose recovery. Our main contributions are: (1) we improve the relocalization ability of a SLAM system using high-level object landmarks; (2) we build an automatic system, capable of identifying, tracking and reconstructing objects with 3D ellipsoids; (3) we show that object-based localization can be used to reinitialize or resume camera tracking. Our fully automatic system allows on-the-fly object mapping and enhanced pose tracking recovery, which we think, can significantly benefit to the AR community. Our experiments show that the camera can be relocalized from viewpoints where classical methods fail. We demonstrate that this localization allows a SLAM system to continue working despite a tracking loss, which can happen frequently with an uninitiated user. Our code and test data are released at gitlab.inria.fr/tangram/oa-slam.Comment: ISMAR 202

View synthesis for pose computation

International audienceGeometrical registration of a query image with respect to a 3D model, or pose estimation, is the cornerstone of many computer vision applications. It is often based on the matching of local photometric descriptors invariant to limited viewpoint changes. However, when the query image has been acquired from a camera position not covered by the model images, pose estimation is often not accurate and sometimes even fails, precisely because of the limited invariance of descriptors. In this paper, we propose to add descriptors to the model, obtained from synthesized views associated with virtual cameras completing the covering of the scene by the real cameras. We propose an efficient strategy to localize the virtual cameras in the scene and generate valuable descriptors from synthetic views. We also discuss a guided sampling strategy for registration in this context. Experiments show that the accuracy of pose estimation is dramatically improved when large viewpoint changes makes the matching of classic descriptors a challenging task

Refining the 3D surface of blood vessels from a reduced set of 2D DSA images

International audienceNumerical simulations, such as blood flow or coil deployment in an intra-cranial aneurism, are very sensitive to the boundary conditions given by the surface of the vessel walls. Despite the undisputable high quality of 3D vascular imaging modalities, artifacts and noise still hamper the extraction of this surface with enough accuracy. Previous studies took the a priori that a homogeneous object was considered to make the reconstruction from the Xray images more robust. Here, an active surface approach is described, that does not depend on any particular image similarity criterion and grounds on high speed computation of the criterion derivatives. Mean square error and normalized cross-correlation are used to successfully demonstrate our algorithm on real images acquired on an anthropomorphic phantom. Preliminary results of coil deployment simulation are also given

Camera Pose Estimation with Semantic 3D Model

International audienceIn computer vision, estimating camera pose from correspondences between 3D geometric entities and their projections into the image is a widely investigated problem. Although most state-of-the-art methods exploit simple primitives such as points or lines, and thus require dense scene models, the emergence of very effective CNN-based object detectors in the recent years have paved the way to the use of much lighter 3D models composed solely of a few semantically relevant features. In that context, we propose a novel model-based camera pose estimation method in which the scene is modeled by a set of virtual ellipsoids. We show that 6-DoF camera pose can be determined by optimizing only the three orientation parameters, and that at least two correspondences between 3D ellipsoids and their 2D projections are necessary in practice. We validate the approach on both simulated and real environments

Camera Relocalization with Ellipsoidal Abstraction of Objects

International audienceWe are interested in AR applications which take place in man-made GPS-denied environments, as industrial or indoor scenes. In such environments, relocalization may fail due to repeated patterns and large changes in appearance which occur even for small changes in viewpoint. We investigate in this paper a new method for relocalization which operates at the level of objects and takes advantage of the impressive progress realized in object detection. Recent works have opened the way towards object oriented reconstruction from elliptic approximation of objects detected in images. We go one step further and propose a new method for pose computation based on ellipse/ellipsoid correspondences. We consider in this paper the practical common case where an initial guess of the rotation matrix of the pose is known, for instance with an inertial sensor or from the estimation of orthogonal vanishing points. Our contributions are twofold: we prove that a closed-form estimate of the translation can be computed from one ellipse-ellipsoid correspondence. The accuracy of the method is assessed on the LINEMOD database using only one correspondence. Second, we prove the effectiveness of the method on real scenes from a set of object detections generated by YOLO. A robust framework that is able to choose the best set of hypotheses is proposed and is based on an appropriate estimation of the reprojection error of ellipsoids. Globally, considering pose at the level of object allows us to avoid common failures due to repeated structures. In addition, due to the small combinatory induced by object correspondences, our method is well suited to fast rough localization even in large environments

Single View Augmentation of 3D Elastic Objects

International audienceThis paper proposes an efficient method to capture and augment highly elastic objects from a single view. 3D shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and resolve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or ressort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method which makes use of a me chanical model and is able to handle highly elastic objects. Our method is formulated as a energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. The only parameter involved in the method is the Young's modulus where we show in experiments that a rough estimate of its value is sufficient to obtain a good reconstruction. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of our approach. Experiments in the context of minimally invasive liver surgery are also provided

Deformation-based Augmented Reality for Hepatic Surgery

International audienceIn this paper we introduce a method for augmenting the laparoscopic view during hepatic tumor resection. Using augmented reality techniques, vessels, tumors and cutting planes computed from pre-operative data can be overlaid onto the laparoscopic video. Compared to current techniques, which are limited to a rigid registration of the pre-operative liver anatomy with the intra-operative image, we propose a real-time, physics-based, non-rigid registration. The main strength of our approach is that the deformable model can also be used to regularize the data extracted from the computer vision algorithms. We show preliminary results on a video sequence which clearly highlights the interest of using physics-based model for elastic registration

Handling Occlusion in Augmented Reality Systems: A Semi-Automatic Method

We present a semi-automatic approach to solve occlu- sion in AR systems. Once the occluding objects have been segmented by hand in selected views called key-frames, the occluding boundary is computed automatically in the in- termediate views. To do that, the 3D reconstruction of the occluding boundary is achieved from the outlined silhou- ettes. This allows us to recover a good prediction of the 2D occluding boundary which is refined using region-based tracking and active contour models. As a result, we get an accurate estimation of the occluding objects. Various results are presented demonstrating occlusion resolution on real video sequences. Results and videos are available at the URL: http://cvlab.epfl.ch/~lepeti